Large receive offload for virtual machines

ABSTRACT

A network interface controller (NIC) that includes a set of receive NIC queues capable of performing large receive offload (LRO) operations by aggregating incoming receive packets is provided. Each NIC queue turns on or off its LRO operation based a set of LRO enabling rules or parameters, whereby only packets that meet the set of rules or parameters will be aggregated in the NIC queue. Each NIC queue is controlled by its own set of LRO enabling rules such that the LRO operations of the different NIC queues can be individually controlled.

BACKGROUND

Large Receive Offload (LRO) has become a feature on almost all networkadapters or network interface controllers (NICs). This feature istypically turned on for end nodes terminating TCP traffic in order toget a boost in throughput to the application terminating the connectionon that node. However, blindly forwarding a large, LRO aggregated packetwould require downstream fragmentation of packets, leading toperformance degradation. Furthermore, traffic being forwarded out of ahost machine must comply with Maximum Segment Size (MSS), but MSS is aparameter that is visible only on the TCP layer and not available to aforwarding VM. Performing LRO aggregation on forwarded traffic wouldtherefore likely to create oversized packets that exceed the MSSrequirement and results in fragmentation.

In most NICs, LRO is a Boolean feature that is simply turned on or off.However, a host machine in a network virtualization environment can hostone or more virtual machines (VMs), some of which may be forwardingtraffic rather than terminating traffic. In some host machines, a VM mayterminate some types of traffic while forwarding other types of traffic.In order to avoid fragmentation of packets on forwarded traffic, manyhost machines in network virtualization environment simply elect to turnoff the LRO feature in the NIC.

What is needed is a host machine that is able to fully utilize the LROcapability of its NIC for maximizing throughput and performance. Such ahost machine should be able to enable LRO aggregation on traffic beingterminated by a VM while disabling LRO aggregation on traffic beingforwarded by a VM. Such a host machine should also be able to maximizethroughput even on forwarded traffic by LRO aggregation without causingunnecessary fragmentation downstream by violating the MSS requirement.

SUMMARY

Some embodiments of the invention provide a network interface controller(NIC) that includes a set of receive NIC queues capable of performinglarge receive offload (LRO) operations by aggregating incoming receivepackets. In some embodiments, each NIC queue turns on or off its LROoperation based a set of LRO enabling rules or parameters, whereby onlypackets that meet the set of rules or parameters will be aggregated inthe NIC queue. In some embodiments, each NIC queue is controlled by itsown LRO enabling rule such that the LRO operations of the different NICqueues can be individually controlled.

In some embodiments, the NIC described above is a physical NIC (PNIC).The PNIC has several receive NIC queues, each NIC queue controlled byits own set of LRO enabling rules such that the LRO operations of thedifferent NIC queues can be individually controlled. In someembodiments, at least some of the operations of the PNIC are controlledby a PNIC driver, which in turn provides an application programminginterface (API) to the virtualization software for controlling the LROoperations and other PNIC operations. The API allows the virtualizationsoftware and other software components of the host machine to set theLRO enabling rules of the individual NIC queues in the PNIC.

In some embodiments, a LRO rule for a NIC queue is a destination addressfilter that enables LRO operation for a specific destination address(MAC address, IP address, or other types of destination address). Insome embodiments, the LRO rule for a NIC queue specifies a particular“flow” or “microflow” for which the LRO operation is to be enabled. Insome of these embodiments, the flow is specified by a set of parametersthat specifies a network session or a transport connection (e.g., thefive-tuple parameters of a TCP/IP connection).

Some embodiments perform LRO aggregation on packets being forwarded by aVM. Some of these embodiments segment the LRO aggregated packetaccording to the Maximum Segment Size (MSS) of the TCP protocol beforeforwarding the segmented packets to their destination. Some embodimentssnoop the packets being forwarded for its MSS parameter before using thesnooped MSS parameter to perform Transmit Segmentation Offload (TSO)operation. In some embodiments, the segmentation operation that uses theextracted MSS parameter is performed by a PNIC of the host machine. Insome of these embodiments, the PNIC performs both the aggregationoperation (LRO) and the segmentation (TSO) within its own hardwarewithout consuming CPU cycles at the host machine. In some embodiments,the PNIC receives the MSS parameter from the network stack as a metadatathat accompanies a LRO aggregated packet.

The preceding Summary is intended to serve as a brief introduction tosome embodiments of the invention. It is not meant to be an introductionor overview of all inventive subject matter disclosed in this document.The Detailed Description that follows and the Drawings that are referredto in the Detailed Description will further describe the embodimentsdescribed in the Summary as well as other embodiments. Accordingly, tounderstand all the embodiments described by this document, a full reviewof the Summary, Detailed Description and the Drawings is needed.Moreover, the claimed subject matters are not to be limited by theillustrative details in the Summary, Detailed Description and theDrawings, but rather are to be defined by the appended claims, becausethe claimed subject matters can be embodied in other specific formswithout departing from the spirit of the subject matters.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth in the appendedclaims. However, for purpose of explanation, several embodiments of theinvention are set forth in the following figures.

FIG. 1 illustrates a NIC having NIC queues whose LRO operations can beindividually enabled or disabled.

FIG. 2 conceptually illustrates a process for setting LRO rules to NICqueues individually.

FIG. 3 illustrates a host machine with a PNIC having NIC queues whoseLRO operations are controlled by API of the PNIC's driver.

FIG. 4 illustrates a host machine operating a software forwardingelement and network stacks between a PNIC and VMs.

FIG. 5 conceptually illustrates the handling of LRO aggregated packetsby a network stack for a VM in a host machine.

FIG. 6 illustrates a host machine that is running virtualizationsoftware that performs LRO aggregation.

FIG. 7 illustrates a virtualization software in which LRO aggregationtakes place after packet forwarding by the software forwarding element.

FIG. 8 illustrates a host machine that assigns packets for different VMsinto a same queue.

FIG. 9 illustrates the assignment of packets for a same VM intodifferent queues.

FIG. 10 illustrates different queues that are each enabled to performLRO aggregation under different five-tuple filtering.

FIG. 11 illustrates LRO aggregation rules applied to queues that areassigned to computing resources in a host machine.

FIG. 12 illustrates LRO aggregation rules applied to NIC queues orincoming packet buffers that are each associated with a VM.

FIG. 13 illustrates an LRO aggregation rule applied to a queue that isnot bound to any specific VMs.

FIG. 14 illustrates an LRO aggregation rule that specifies only adestination address for a queue in a host machine.

FIG. 15 illustrates a host machine in which LRO aggregation rules arebeing applied across different NIC queues or incoming packet buffers.

FIG. 16 conceptually illustrates a process for applying LRO aggregationrules to packets in NIC queues or incoming packet buffers.

FIGS. 17a-b conceptually illustrates the snooping of MSS parameter andthe use of the snooped MSS parameter for packet segmentation on packetsbeing forwarded by a VM.

FIG. 18 illustrates the snooping and extraction MSS parameter at layersof network stack lower than TCP layer.

FIG. 19 illustrates the segmentation (TSO) operation by a PNIC based ona MSS parameter extracted by a network stack in a host machine.

FIG. 20 conceptually illustrates the maintenance of stateful tables by aforwarding VM while snooping for MSS.

FIG. 21 conceptually illustrates a process for snooping MSS parameterand a process for performing segmentation on LRO aggregated packetsbased on the snooped MSS parameter.

FIG. 22 conceptually illustrates an electronic system with which someembodiments of the invention are implemented.

DETAILED DESCRIPTION

In the following description, numerous details are set forth for thepurpose of explanation. However, one of ordinary skill in the art willrealize that the invention may be practiced without the use of thesespecific details. In other instances, well-known structures and devicesare shown in block diagram form in order not to obscure the descriptionof the invention with unnecessary detail.

Some embodiments of the invention provide a network interface controller(NIC) that includes a set of receive NIC queues capable of performinglarge receive offload (LRO) operations by aggregating incoming receivepackets. In some embodiments, each NIC queue turns on or off its LROoperation based a set of LRO enabling rules or parameters, whereby onlypackets that meet the set of rules or parameters will be aggregated inthe NIC queue. In some embodiments, each NIC queue is controlled by itsown LRO enabling rule such that the LRO operations of the different NICqueues can be individually controlled.

In some embodiments, LRO is turned on or off in the hardware or softwarebased upon flow information programmed on the component performing theLRO aggregation. The flow or micro flow could be L2 or L3-L4 flow withwildcards. Hence for example, some embodiments enable LRO for alltraffic destined to a particular VM's MAC address. This way the VM andvirtualization software can gain in efficiency and throughput as thisreduces the number of packets hitting through the VM.

FIG. 1 illustrates a NIC 120 having NIC queues 121-124 whose LROoperations can be individually enabled or disabled. The NIC 120 is partof a host machine 100 in a network virtualization environment. The hostmachine 100 is communicatively linked to other network nodes through toa physical network 190 and hosting virtual machines (VMs) 111-114. Thehost machine also provides network transmit/receive processing 140between the NIC 120 and the VMs 111-114.

The host machines 100, in some embodiments, is a computing devicemanaged by an operating system (e.g., Linux) that is capable of creatingand hosting VMs. The host machine provides the computing resources (suchas CPU cores and memories) needed for performing the computing tasks ofthe VMs. The host machine also provides the network communicationresources needed for allowing each VM to participate in the networktraffic of the physical network 190.

The VMs 111-114 are virtual machines operating in the host machine 100.The VMs executes on top of a hypervisor (not shown), which, in someembodiments, includes the network virtualization layer. Networkvirtualization will be further described by reference to FIG. 3 below.In some embodiments, the VMs 111-114 are each assigned a set of networkaddresses (e.g., a MAC address for L2, an IP address for L3, etc.) andcan send and receive network data to and from other network elements,such as other VMs. In some embodiments, at least some of the VMs act astraffic terminals of network traffic that generates or consume networktraffic. In some embodiments, at least some of the VMs act as forwardingelements of network traffic that forward received data packets on toother elements in the network. In the example host machine illustratedin FIG. 1, the VMs 112 and 113 are operating as forwarding elements,while the VMs 111 and 114 are operating as network traffic terminals.

In some embodiments, a forwarding VM both forwards and consumes (i.e.,uses) the network traffic that it receives. In some embodiments, aforwarding VM replicates the received network traffic to multiplerecipients. In some embodiments, a forwarding VM does not use or consumethe network traffic that it receives and just forwards. In some of theseembodiments, the host machine is operating a multi-layered network stackfor each VM, and a forwarding VM performs packet forwarding at lowerlevels of the network stack (e.g. below TCP level). In some embodiments,a forwarding VM is an instance of a logical routing element (LRE) thatperforms L3 routing between different IP subnets. Description of LREscan be found in U.S. patent application Ser. No. 14/137,862, filed Dec.20, 2013, now published as U.S. Patent Publication 2015/0106804.

The host machine 100 performs various functions and operations betweenthe NIC 120 and the VMs 111-114. These functions and operations arecollectively referred to as the network TX and RX processing 140 inFIG. 1. Different embodiments implement the TX and RX process 140differently. Some embodiments implement the network TX and RX processing140 as one module within the host machine 100, while some otherembodiments implement the network TX and RX processing 140 in severalmodules.

In some embodiments, the TX and RX processing 140 includes networkstacks for the VMs 111-114. In some embodiments, at least some of thenetwork stacks are implemented according to a multi-layer networkingmodel such as TCP/IP. In some embodiments, each of the network stacksincludes an I/O chain (not illustrated) that performs the layerednetwork operations as well as other operations. Network stacks will befurther described by reference to FIGS. 4-5 below.

In some embodiments, the TX and RX processing 140 includes L2 switchingoperations and/or L3 routing operations. In some of these embodiments,these switching and/or routing operations are performed by an instanceor instances of software forwarding elements such as logical forwardingelements (LFEs), logical switching elements (LSEs), or aforementionedLREs. Software forwarding elements will be further described byreference to FIG. 4.

The NIC module 120 is the interface to the physical network 190 for thehost machine 100. As illustrated, the NIC module 120 includes a numberof queues 121-124 for queuing incoming network traffic packets from thephysical network 190. The NIC module also includes a queue assignmentsub-module 129 for assigning incoming data packets to the queues121-124. In some embodiments, each of the NIC queues is assigned to acomputing resource (e.g., processor core or a processing thread) in thehost machine 100, and the queue assignment sub-module 129 assignsincoming packet to the queues according to the computing resourceassignment. In some embodiments, each of the NIC queues is associatedwith a VM. In some of these embodiments, each VM is addressable by a MACaddress or IP address, and the queue assignment sub-module 129 filtersincoming network packets into the queues based on the MAC address or theIP address of the VMs.

As illustrated, the NIC queues 121-124 are associated with the VMs111-114, respectively. In some embodiments, a VM is associated with aNIC queue because the NIC queue receives only data packet filtered forthat VM. In some embodiments, a VM is associated with a NIC queuebecause the NIC queue is assigned to a processor core or a CPU that isrunning the network stack or the processing threads of the VM. In someembodiments, a NIC queue is not associated with any particular VM, butreceived data packets will be distributed to their correct destinationVMs according to their destination IP address or MAC address.

The NIC queues 121-124 are for buffering incoming data packets receivedfrom the physical network 190. Furthermore, each of the NIC queues121-124 is capable of performing LRO operations, i.e., aggregatingsmaller incoming network packets into larger data packets for efficientprocessing and delivery to the VMs. The LRO operations of each NIC isindividually controlled by a LRO rule or a set of LRO rules for that NICqueue, and the NIC queue enables LRO operations on packets that complywith the LRO rule for that queue (i.e., aggregates a smaller data packetin the NIC queue into a aggregated LRO packet when the smaller datapacket meets the criteria set forth by the LRO rule). In someembodiments, such rules are supplied by a network controller 170 thatcontrols the networking operations of the host machines (include 100)that are connected to the physical network 190.

In some embodiments, a LRO rule for a NIC queue is a destination addressfilter that enables LRO operation for a specific destination address(MAC address, IP address, or other types of destination address). Insome embodiments, the LRO rule for a NIC queue specifies a particular“flow” or “microflow” for which the LRO operation is to be enabled. Insome of these embodiments, the flow is specified by a set of parametersthat specifies a network session or a transport connection (e.g., thefive-tuple parameters of a TCP/IP connection). In other words, the LROrule specifies that packet aggregation is to be turned on for datapackets of a particular network session or connection but not forothers. Flow-based LRO control will be further described in Section IIbelow.

As mentioned, in some embodiments, the LRO operations of the differentNIC queues are individually enabled or disabled. In other words, LROoperations can be enabled for some NIC queues while disabled for others.FIG. 1 illustrates the individual enable/disable of the LRO operationsin queues 121-124 of the NIC 120. In the NIC 120, the LRO operations ofthe NIC queue 121-124 are controlled by LRO control modules 131-134,respectively. As illustrated, the LRO control 131 receives a LRO rulethat enables LRO aggregation in the NIC queue 121, while the LRO control132 receives a different LRO rule that disables LRO aggregation in theNIC queue 122. The NIC queues 123 and 124 are likewise respectivelydisabled and enabled by each's own LRO controls 133 and 134. Asmentioned, some embodiments enable LRO aggregation for a particulardestination VM, hence all packets in the NIC queue of the particulardestination VM will be aggregated by LRO operation. For some embodimentsthat enable LRO for a particular flow (e.g., a TCP connection), onlypackets in the NIC queue that belongs to that particular flow will beaggregated.

By allowing the LRO operations of NIC queues to be individually enabledor disabled, the host machine in some embodiments allows effectivecontrol of LRO operations for different VMs. For example, in someembodiments, it is desirable to turn on LRO operations for trafficterminating at a VM while turning off LRO operations for traffic thatare to be forwarded. This is at least partly because packets that are tobe forwarded out of the host machine must comply with a maximum sizerequirement, and hence it is desirable to turn off LRO operations forthose VMs (e.g., VMs 112 and 113) that are forwarding packets out of thehost machine. This avoids creating LRO aggregated packets that mayexceed the maximum size limit (e.g., Ethernet MTU), which requiredownstream segmentation operations (e.g., TSO operations) orfragmentation to dissolve the aggregated packets into smaller segmentsor fragments. On the other hand, for VMs that consume the incomingnetwork packets and do not forward those packets (e.g., the VMs 111 and114), it is advantageous to perform LRO aggregation for reducingoverhead, because LRO aggregated packets that exceed the maximum sizerequirement of the physical network would not need to be segmented orfragmented later. Some embodiments allow further effective control ofthe LRO operations by enabling and disabling LRO operations in each NICqueue on a connection by connection, or session by session basis.

FIG. 2 conceptually illustrates a process 200 for setting LRO rules toNIC queues individually. The process 200 starts when it receives (at210) a configuration for the host machine. In some embodiments, such aconfiguration specifies the VMs that will be running on the hostmachine. In some embodiments that operate a virtualization software inthe host machine, such configuration also specifies parameters forcomponents of the virtualization software such as software switchesand/or software routers.

The process then identifies (at 220) a set of LRO rules for the hostmachine. In some embodiment, such LRO rules are identified in order tooptimize the performance of the host machine. For example, someembodiments set the LRO rules so that packets that will be consumed by aVM within the host machine will be aggregated while packets that will beforwarded by a VM would not be aggregated. This is done to minimizeoverhead in processing small packets and to avoid having to segment orfragment oversized packet. Some embodiments identify transportconnections that are forwarded or consumed by VMs in the host machine.The process then enables LRO aggregation for transport connections thatterminate at VMs in this host machine while disable LRO aggregation fortransport connections that are to be forwarded by VMs in this hostmachine.

Next, the process identifies (at 230) NIC queues that are needed forimplementing those LRO rules. In some embodiments, an LRO rule can befor packets destined for a particular VM, so to perform LRO on thosepackets requires identifying the NIC queues that buffers the packets forthat particular VM. In some embodiments, an LRO rule can be for aparticular type of packet (e.g., of a particular TCP connectionidentifiable by a five-tuple), and the process would identify a NICqueue that is assigned to hold that particular type of packets.

Once a NIC queue is identified for a particular LRO rule, the processapplies (at 240) the particular LRO rule to the identified NIC queue. Inthe example of FIG. 1, this operation includes sending the LRO rule tothe LRO control module of the identified NIC queue. In some embodimentsin which the NIC is a physical NIC (PNIC), the operation includessetting the LRO rules at the queues of the PNIC through the API of thedriver of the PNIC.

The process then determines (at 250) if there are other LRO rules toimplement in the NIC queues. If so, the process returns to 230 toidentify a NIC queue for another LRO rule. If not, the process 200 ends.

The solution described above allows effective turning on of LRO on microflow level so that traffic destined to a particular VM can have LROenabled in hardware or software. This allows the virtualization softwareto efficiently use the resources available to it.

Several more detailed embodiments of the invention are described below.Section I describes individually enabled NIC queues in a physical NIC.Section II describes controlling LRO operations using various LROaggregation rules. Section III describes snooping maximum segment sizeparameter from forwarded packets. Finally, section IV describes anelectronic system with which some embodiments of the invention areimplemented.

I. Individually Controlled LRO in a Physical NIC

In some embodiments, the NIC described above is a physical NIC (PNIC) ina host machine of a virtualized network environment. The PNIC is anetwork adaptor that has dedicated network interfacing hardware forprocessing incoming and outgoing network traffic without consumingprocessor (i.e., CPU) cycles of the host machine. The host machineoperates virtualization software, which allows multiple VMs tosimultaneously operate in the host machine and to have network accessthrough the PNIC to the physical network. The PNIC has several receiveNIC queues, each NIC queue controlled by its own set of LRO enablingrules such that the LRO operations of the different NIC queues can beindividually controlled. In some embodiments, at least some of theoperations of the PNIC are controlled by a PNIC driver, which in turnprovides an application programming interface (API) to thevirtualization software for controlling the LRO operations and otherPNIC operations. The API allows the virtualization software and othersoftware components of the host machine to set the LRO enabling rules ofthe individual NIC queues in the PNIC.

FIG. 3 illustrates a host machine 300 with a PNIC 320 having NIC queueswhose LRO operations are controlled by the API of the PNIC's driver. TheAPI makes the control of the individual NIC queues in the PNIC availableto virtualization software running in the host machine. As illustrated,the host machine 300 is running virtualization software 340, whichallows the host machine 300 to host VMs 311-314. The virtualizationsoftware 340 also provides network virtualization to the VMs 311-314,which allows the VMs to send and receive network traffic from thephysical network 390 through the PNIC 320. The control of the PNIC 320is available to the host machine 300 through the PNIC driver 360, whichprovides API 365 as interface for software components (including thevirtualization software 340) running on the host machine 300. In someembodiments, these various software components are program(s) beingexecuted on one or more processors or processor cores in the hostmachine. The software components running on the host machine invokes APIroutines of the PNIC driver 360, which cause the driver to issuecommands or send data to the PNIC 320, or to retrieve data or handleinterrupt from the PNIC 320.

The PNIC 320 is a physical hardware component dedicated to performingthe function of a NIC. In some embodiments, the host machine 300offloads network interfacing tasks from its CPUs/processors to the PNIC320. As illustrated, the PNIC 320 includes receive queues 321-324 forbuffering incoming packets from the physical network 390, each of thesequeues capable of LRO packet aggregation operations as described above.The LRO operations of each of these queues 321-324 are controlled by itsown LRO control module (331-334, respectively). In some embodiments, thequeuing of incoming network traffic as well as the aggregating of datapackets (LRO) are handled by the PNIC 320 without consumingCPU/processor cycles in the host machine 300. In some of theseembodiments, the LRO rules to the different NIC queues are provided bythe virtualization software 340 through the API 365, while the PNIC 320uses the API 365 to inform the virtualization software 340 that aparticular NIC queue has completed aggregating a packet under theparticular NIC queue's LRO aggregation rule. In some embodiments, thevirtualization software in turn fetches the received data packets fromthe PNIC, either aggregated (if LRO is turned on) or not aggregated (ifLRO is turned off). As illustrated, the API 365 also allows LROoperations to be controlled by an external network controller 370, whichin some embodiments pushes down configuration or control data thatincludes rules for LRO aggregation to the host machine 300.

The PNIC 320 also includes a queue assignment sub-module 329 and a RSS(receive side scaling) sub-module 328. The queue assignment sub-module329 determines to which NIC queue does an incoming data packet from thephysical network goes into, while the RSS sub-module assigns each NICqueue to a computing resource (e.g., processor core or processingthread) in the host machine. In some embodiments, the RSS 328 assignsnetwork traffic to processing threads, and the threads can be assignedto different CPUs for load balancing purposes. In some of theseembodiments, each VM is associated with a processing thread, and the RSS328 ensures that a thread of a VM stays on a CPU for the duration of anetwork connection session. In some embodiments, the queue assignmentsub-module 329 assigns packets to the different queues by simply hashingthe incoming packets (e.g., by hashing on the destination MAC address orIP address), the RSS 328 sub-module in turn distributes packets from theNIC queues based on the same hashing function.

In some embodiments, each of the NIC queues 321-324 is directlyassociated with a VM, and the queue assignment sub-module 329 filtersincoming network packets into the queues based on the MAC address or theIP address of the VMs. In some embodiments, a NIC queue in the PNIC 320is not associated with any particular VM, but the received data packetswill be distributed to their correct destination VMs according to theirdestination IP address or destination MAC address by a softwareforwarding element. In some embodiments, the queue assignment sub-module329 filters incoming network packets into the queues based on flows ormicro-flows that are associated with connection sessions. For example, aNIC queue can have a filter that accepts only packets having aparticular five-tuple identifier belonging to a particular TCPconnection.

In some embodiments, the queue assignment sub-module 329 assigns packetsfrom multiple different VMs to a same NIC queue. This is the case forsome host machines operate more VMs than NIC queues such that at leastsome of the NIC queues necessarily serve multiple VMs. A host machinethat operates more VMs than NIC queues will be further described belowby reference to FIG. 8. In some embodiments, the queue assignmentsub-module 329 assigns packets for a particular VM to only oneparticular NIC queue such that all of the packets heading to thatparticular VM will be subject to a same set of LRO rules being appliedto that particular queue. In some embodiments, the queue assignmentsub-module 329 can assign packets for a same VM across different queues.This can occur if the queue assignment sub-module uses criteria otherthan destination address for assigning packets to a queue (e.g., byhashing or by connection session identified by five-tuple). Assigningpackets for a same VM across different NIC queues will be furtherdiscussed by reference to FIG. 9 below.

As discussed above, different embodiments assign incoming packets toqueues based on different types of criteria (MAC address, five-tuple,simple hashing, etc.). In some embodiments, different queues in a NICcan be programmed to accept packets based on different types ofcriteria, i.e., a NIC can use a mixture of types of criteria forassigning packets into queues. For example, in some embodiments, a NICcan have some queues that use MAC filters for accepting incomingpackets, some queues that use connection/five-tuple filters, while otherqueues receive packets solely based on a hashing function.

Though not illustrated, a PNIC in some embodiments have different typesof NIC queues, in which some of the NIC queues are assigned to computingresources, while others are assigned to corresponding destination VMs bysetting MAC filters. In some embodiments, some of the queues in the PNICare dedicated to specialized hardware functions. For example, some ofthe NIC queues have specialized hardware support for performing LROoperations, and can be programmed to perform LRO aggregation for anydestination VM.

The PNIC 320 also includes a command/data interface 327 for handling thecommunication between the PNIC 320 and the processor core of the hostmachine 300. The PNIC driver 360 communicates with the PNIC 320 whenroutines of the API 365 are invoked at the processor core of the hostmachine 300 (e.g., by the virtualization software 340). The command/datainterface 327 translates signals received from the process core intodata packets for the network, or into control signals for variouscomponents of the PNIC 320. Among these control signals are the LROaggregation rules for each of the NIC queues 321-324, where differentLRO control modules 331-334 are controlled by different control signalsfrom the command/data interface 327. In some embodiments, the processoris able to implement a LRO aggregation rules at a particular NIC queuesby invoking an API routine that addresses the control signals of aparticular LRO control module.

In some embodiments, the command/data interface 327 also supportcommunication with the driver 360 by interrupt or by polling. Forexample, when there is an aggregated packet ready for delivery to one ofthe VMs from one of the NIC queues 321-324, the command/data interface327 updates a set of corresponding status bits so the processor wouldknow that there is a packet ready for retrieval when it polls the statusbits. The processor then invokes an API routine to retrieve the packetfrom the PNIC 320 through the command/data interface 327.

The virtualization software 340 manages the VMs 311-314. Virtualizationsoftware may include one or more software components and/or layers,possibly including one or more of the software components known in thefield of virtual machine technology as virtual machine monitors (VMMs),hypervisors, or virtualization kernels. Because virtualizationterminology has evolved over time and has not yet become fullystandardized, these terms do not always provide clear distinctionsbetween the software layers and components to which they refer. As usedherein, the term, “virtualization software” is intended to genericallyrefer to a software layer or component logically interposed between avirtual machine and the host platform.

In some embodiments, the virtualization software 340 assigns thecomputing resources of the host machine (e.g., CPU cycles) to the VMs311-314. In some embodiments, the virtualization software also conductsnetwork traffic between the PNIC 320 and the VMs 311-314 as well asamong the VMs 311-314 themselves. In some of these embodiments, thevirtualization software includes one or more software forwarding elementfor forwarding data packets to and from the VMs in the host machine. Inaddition, the host machine also operates a network stack or protocolstack for each of the VMs. For some of these embodiments, FIG. 4illustrates the host machine 300 operating a software forwarding elementand network stacks between the PNIC 320 and the VMs 311-314.

As illustrated in FIG. 4, in addition to the PNIC driver 360 and thePNIC 320, the host machine 300 is operating a software forwardingelement 440 and network stacks 411-414 for the VMs 311-314. In someembodiments, the software forwarding element 440 and the network stacks411-414 are part of the virtualization software running on the hostmachine 300 (e.g., the virtualization software 340). Each network stackconnects to its VM and the software forwarding element 440, which isshared by all the network stacks of all the VMs. Each network stackconnects to the software forwarding element 440 through a port of thesoftware forwarding element. In some embodiments, the softwareforwarding element 440 maintains a single port for each VM.

The software forwarding element 440 connects to the PNIC 320 through thePNIC driver 360 to send outgoing packets and to receive incomingpackets. In some embodiments, the software forwarding element is definedto include an uplink through which it connects to the PNIC to send andreceive packets. The software forwarding element 440 performspacket-processing operations to forward packets that it receives on oneof its ports to another one of its ports, or through the uplink and thephysical network to another host machine. For example, in someembodiments, the software forwarding element 440 tries to use data inthe packet (e.g., data in the packet header) to match a packet to flowbased rules, and upon finding a match, performs the action specified bythe matching rule.

In some embodiments, software forwarding elements executing on differenthost devices (e.g., different computers) are configured to implementdifferent logical forwarding elements (LFEs) for different logicalnetworks of different tenants, users, departments, etc. that use thesame shared computing and networking resources. For instance, twosoftware forwarding elements executing on two host devices can performL2 switch functionality. Each of these software switches can in partimplement two different logical L2 switches, with each logical L2 switchconnecting the VMs of one entity. In some embodiments, the softwareforwarding elements provide L3 routing functionality, and can beconfigured to implement different logical routers with the software L3routers executing on other hosts.

In the virtualization field, some refer to software forwarding elementsas virtual forwarding elements as these are software elements. However,in some embodiments, the software forwarding elements are referred to asphysical forwarding elements (PFEs), in order to distinguish them fromlogical forwarding elements (LFEs), which are logical constructs thatare not tied to the physical world. In other words, the softwareforwarding elements are referred to as PFEs because they exist andoperate in the physical world, whereas logical forwarding elements aresimply a logical representation of a forwarding element that ispresented to a user. Examples of software forwarding elements such assoftware switches, software routers, etc. can be found in U.S. patentapplication Ser. No. 14/137,862.

Each network stack processes network traffic from the PNIC 320 to itscorresponding VM across the different layers of network protocols. Insome embodiments, this includes handling network protocols at link layer(e.g., Ethernet or MAC), network layer (e.g., IP), transport layer(e.g., TCP), and/or application layer (e.g., HTTP). In some embodiments,one or more of the layered protocols of a network stack is handled bythe corresponding VM.

In some embodiments, an LRO aggregated packet is a TCP layer packet(i.e., having TCP headers and specifying a TCP port as destination).Such an LRO aggregated packet destined for a particular VM is processedat the transport layer of the particular VM's network stack according tothe TCP protocol. In some embodiments, the network stack communicateswith the PNIC through the PNIC's API in order to retrieve LRO aggregatedpackets from the NIC queue.

FIG. 5 conceptually illustrates the handling of LRO aggregated packetsby the network stack 411 for the VM 311 in the host machine 300. Thenetwork stack 411 includes a link layer 501, an internet layer 502, atransport layer (TCP) layer 503, and an application layer 504. Thenetwork stack communicates with the NIC queue 321 in the PNIC 320through the API 365 of the PNIC driver 360. The NIC queue 321 hasfinished aggregating a packet 510 under a LRO rule 590. The aggregatedLRO packet 510 has TCP header for processing at TCP layer.

FIG. 5 illustrates the retrieval of a LRO packet through the PNIC's APIin four operations labeled ‘1’ through ‘4’. At the operation labeled‘1’, the PNIC 320 detects that an LRO aggregated packet is ready fordelivery in the NIC queue 321 and sends an interrupt to the processorcore, which communicates with the PNIC through the PNIC driver 360 usingthe PNIC's API 365. At the operation labeled ‘2’, the TCP layer of thenetwork stack picks up the interrupt signal and recognized that there isan LRO packet sitting in the PNIC 320. At the operation labeled ‘3’, theaggregated packet 510 is retrieved from the NIC queue 321 by using theAPI 365. The packet 510 traverses through lower layers of the networkstack 411 (i.e., link layer 501 and IP layer 502) before reaching TCPlayer 503. At the operation labeled ‘4’, the TCP layer 503 processes theLRO packet according to TCP protocol and passes the content of the LROaggregated packet to the application layer 504 and the VM 311.

In some embodiments, each network stack is operated by processingthreads running on the host machine's processor core(s). In some ofthese embodiments, each thread manages a queue in the PNIC 320. Whenevera queue in the PNIC has a packet ready (e.g., a LRO packet) for deliveryto the corresponding network stack, the PNIC 320 generates an interruptto the processor core that executes the network stack's processingthread. The PNIC 320 sends this interrupt through the API 365, which inturn passes the interrupt to the processor core. In some embodiments,each time a queue's thread is invoked for this operation, the core thatmanages the queue and executes its thread has to interrupt another taskthat it is performing to execute the thread so that it can retrieve thepackets from the queue. Such interruptions affect the processor'soperational efficiency. By performing LRO operations and aggregatingmany small packets into fewer, larger packets, some embodiments increasethe operational efficiency of the processor by reducing the number ofinterrupts that the processor has to handle.

In some embodiments, LRO aggregation is not implemented in the PNIC, butis instead implemented within the virtualization software. FIG. 6illustrates a host machine 600 that is running virtualization software605 that performs LRO aggregation. The virtualization software 605 isoperating VMs 611-614 and receiving data packets from a physical network690 through a PNIC 607.

The virtualization software 605 includes network stacks 621-624 for theVMs 611-614, respectively. The virtualization software 605 also includesa software forwarding elements 630 for forwarding packets between thePNIC and the VMs. The software forwarding element 630 has several ports,each port of the virtualization software is connected to a network stack(621-624) of a VM (611-614). The software forwarding element 630receives data packets from the PNIC 607 through a set of queues 641-644.Like the NIC queues in the PNIC as described above by reference to FIGS.3-5, the queues are for buffering incoming data packets received fromthe physical network. And like the NIC queues in the PNIC, each queue inthe set of queues 641-644 is capable of performing LRO aggregationoperation based on its own LRO aggregation rule (651-654). In someembodiments, the LRO aggregation rules are specified by an externalnetwork controller 670. The virtualization software 605 also includes aqueue assignment module 660 for assigning incoming data packets from thePNIC to the queues 641-644.

FIG. 6 illustrates a virtualization software in which the LROaggregation of packets occurs in the queues 641-644 beforepacket-forwarding by the software forwarding element 630. In some otherembodiments, the virtualization software performs LRO aggregation afterpacket forwarding by the software forwarding element. In some of theseembodiments, each VM has its own dedicated LRO aggregation mechanismthat can be turned on or off. FIG. 7 illustrates a virtualizationsoftware in which LRO aggregation takes place after packet forwarding bythe software forwarding element.

FIG. 7 illustrates a host machine 700 that is running virtualizationsoftware 705 that performs LRO aggregation. Like the virtualizationsoftware 605, the virtualization software 705 is operating VMs 711-714and receives data packets from a physical network 790 through a PNIC707. The virtualization software 705 includes network stacks 721-724 forthe VMs 711-714, respectively. The virtualization software 705 alsoincludes a software forwarding elements 730 for forwarding packetsbetween the PNIC 707 and the VMs 711-714. The software forwardingelement 730 has several ports, each port of the virtualization softwareis for forwarding packets to a VM.

Unlike the software forwarding element 630 in the virtualizationsoftware 605, the forwarding element 730 does not receives LROaggregated packets. Rather, each port of the software forwarding element730 forwards received data packet to a queue that is capable ofperforming LRO aggregation before reaching a network stack for a VM. Asillustrated, the queues 741-744 are situated at the ports of thesoftware forwarding element 730, and are for performing LRO aggregationon packets destined to the VMs 711-714, respectively. The network stacks721-724 receives data packets from the queues 741-742, and these datapackets may be LRO aggregated or not based on the LRO aggregation ruleof each of the queues 741-744. Since the LRO capable queues 741-744receive data packets that are already sorted based on destinationaddress, the virtualization software 705 does not include a queueassignment module (like 660) for assigning receive data packets to thosequeues.

In some embodiments, some or all of the queues handle data packets (andhence aggregation) for multiple different VMs. This is particularly thecase when the host machine implementing the NIC queues is operatingfewer NIC queues than VMs. For some embodiments, FIG. 8 illustrates ahost machine 800 that assigns packets for different VMs into a samequeue.

As illustrated in FIG. 8, the host machine 800 is operating eightdifferent VMs 811-818 (VM “A” through “H”). The host machine also hascomputing resources 861-864 for operating the VMs 811-818. The hostmachine also includes a RX processing module 820, four incoming databuffers 831-834, and a queue assignment module 850. In some embodiments,the incoming data buffers 831-834 are NIC queues in a PNIC (notillustrated), and the queue assignment module 850 is part of the PNIC.In some embodiments, the incoming data buffers 831-834 and the queueassignment module 850 are implemented in a virtualization software (notillustrated) running on the host machine 800.

The RX processing module 820 encapsulate functions performed by the hostmachine that retrieve, process, and forward packets to the VMs 811-818.In some embodiments, the RX processing module 820 includes a softwareforwarding element for forwarding packets to the VM as well as networkstacks for processing network protocols for the VMs 811-818. In someembodiments, the RX processing module 820 represents a collection ofsoftware modules performed by the virtualization software running on thehost machine 800.

The queue assignment module 850 receives incoming packets from thephysical network 850 and assigns the received packets to the incomingdata buffers 831-834. As mentioned, different embodiments assignincoming packets to queues based on different types of criteria (MACaddress, five-tuple, simple hashing, etc.). Furthermore, in someembodiments, different queues in a NIC can be programmed to acceptpackets based on a mixture of different types of criteria.

As illustrated, at least some of the incoming data buffers have packetsfrom different queues. Specifically, the queue 831 is assigned packetsfor VMs 811 and 812 (VMs A and B), the queue 832 is assigned packets forVMs 813, 814, and 815 (VMs C, D, and E), the queue 833 is assignedpackets for VM 816 (VM F), and the queue 834 is assigned packets for VM817 and 818 (VMs G and H). In this particular example, the sharing of atleast some of the queues by multiple VMs is necessary because there aremore VMs (eight) than there are queues (four). In some embodiments, eachincoming data buffer or queue is associated with a computing resource(e.g., a thread or a CPU core), and VMs that operates on a samecomputing resource would share a same queue, regardless of whether thereare more VMs than queues. In this example, the VMs 811-813 are operatingon a same computing resource 861, so the data packets for the VMs811-813 are assigned to a same queue 831.

As illustrated, the LRO operations of the queues 831-834 are governed byLRO aggregation rules 841-844, respectively. Thus, the rule 841 governsthe LRO aggregation operations of the VMs 811-812, the rule 842 governsthe LRO aggregation operations of the VMs 813-815, the rule 843 governsthe LRO aggregation operations of the VM 816, and the rule 844 governsthe LRO aggregation operations of the VMs 817-818.

In some embodiments, the assignment of packets to queues is such thatpackets for a particular VM may end up in different queues. This canoccur if the queue assignment is at least partly based on simple hash,or if the queue assignment is at least partly based on flow ormicro-flow (e.g., specified by five tuple identifiers for a TCPconnection session) filtering that does not correspond directly to a VMin the host machine. Consequently, in some of these embodiments, LROrules may be applicable to only some packets of a VM (e.g., belonging toa particular connection session) but not to other packets of the same VM(e.g., not belonging to the particular connection session).

FIG. 9 illustrates the assignment of packets for a same VM intodifferent queues in the host machine 800. The queue assignment module850 of the host machine 800 is applying a different set of filters inFIG. 9 than in FIG. 8. As illustrated, this different set of filterscause the queue assignment module 850 to assign packets for the VM 811to both the queues 831 and 832, packets for the VM 817 to both thequeues 831 and 834.

As illustrated, the queue assignment module 850 is applying a connectionsession filter 961 on the queue 831 and another connection sessionfilter 962 on the queue 832, such that packets for the VM 811 ends up inboth the queue 831 and the queue 832. In contrast, the queue assignmentmodule 850 is applying a MAC filter 963 to the queue 833 that allowsonly packets for the VM 814 to enter the queue 833. The packets for theVM 814 are not distributed across different queues.

Because packets for a same VM can be in different queues, packets forone particular VM are simultaneously aggregated under different LROrules associated with these different queues. In some embodiments, asame set of LRO rules are applied across different queues such thatpackets of a same VM being assigned to different queues may beaggregated under that same set of rules. In some embodiments, multipleLRO rules are applied to one queue such that packets for different VMscan have different LRO aggregation rules even though they share the samequeue. Examples of LRO rules that are applied across different queueswill be further described by reference to FIG. 15 below.

II. LRO Aggregation Rules

As mentioned, LRO aggregation of incoming data packets to VMs of a hostmachine can be turned on or off based on LRO aggregation rules. In someembodiments the LRO aggregation rules are implemented on individual NICqueues of a PNIC as discussed above by reference to FIGS. 3-5. In someembodiments, the LRO aggregation rules are implemented within avirtualization software of a host machine as discussed above byreference to FIGS. 6-7.

In some embodiments, a LRO aggregation rule is implemented as adestination address filter that enables LRO operation for specificdestination address (MAC address, IP address, or other types ofdestination address). In some embodiments, a LRO aggregation rulespecifies a particular “flow” or “microflow” for which LRO operation isto be enabled. In some of these embodiments, the flow is specified by aset of parameters that specifies a network session or connection (e.g.,the five-tuple parameters of a TCP/IP connection). In other words, theLRO rule specifies that packet aggregation is to be turned on for datapackets of a particular network session or transport connection but notfor others.

For instance, some embodiments use the five-tuple IP data in the L3 andL4 packet header to classify the packet payload. The five-tuple datainclude source port identifier, destination port identifier, source IPaddress, destination IP address, and the protocol. Using these fiveidentifiers, some embodiments can selectively turn on or off LROaggregation for IP packets of different types, such as VOIP packet,video packet, audio packet, FTP packet, HTTP packet, HTTPS packet,Remote Desktop packet (PCoIP, VNC, RDP), management packet(authentication, server health monitoring, time synchronization), E-mailpacket (POP3, SMTP), etc.

The examples provided below illustrates how the five tuples can be usedto differentiate web traffic, VoIP, video streaming, remote desktop,management, e-mails, by using the following notation:Protocol-src_ip-dst_ip-src_port-dest_port, with * denoting wildcardmatch. In these examples, it is assumed that that a VM is the clientthat requests the service/data/service from the server.

-   -   Web: TCP-*-*-*-80/443 (80 for HTTP and 443 for HTTPS)    -   VoIP (Skype): TCP-*-*-23399-* or TCP-*-*-*-23399 (incoming and        outgoing traffic)    -   Video Streaming (MMS): TCP-*-*-*-1755    -   Remote Desktop (PCoIP): TCP-*-*-*-4172    -   Authentication (Kerberos): TCP-*-*-*-88    -   E-Mail (POP3): TCP-*-*-*-110

For some embodiments, FIG. 10 illustrates different queues (or incomingpacket buffers) 1001-1003 that are each enabled to perform LROaggregation under different five-tuple filtering. In some embodiments,the queues 1001-1003 are NIC queues inside a PNIC. In some embodiments,these queues are implemented within the virtualization software of thehost machine. Each queue is enabled to perform LRO aggregation under itsown LRO aggregation rule, such that a packet received from a physicalnetwork arriving at a particular queue will be aggregated into a LROpacket only if it meets the criteria set forth in the LRO aggregationrule of the particular queue. As illustrated, the queues 1001-1003 areenabled to perform LRO aggregation under LRO aggregation rule 1011-1013,respectively.

The LRO aggregation rule 1011 is a five-tuple filter that does not haveany wild cards. It specifies that LRO operation would only be performedfor packets that come from a specific sender (source IP 192.168.10.2)and to a specific recipient (destination IP 10.10.3.1). It furtherspecifies that LRO operation would only be performed for packets of aparticular transport connection (source transport port 1111 anddestination transport port 2222), and that the transport protocol isTCP. Consequently, the queue 1012 would accumulate only data packetswith headers having the five-tuple ofTCP-192.168.10.2-10.10.3.1-1111-2222. All other packets arriving at thequeue 1001 will not be aggregated into an LRO packet.

The LRO aggregation rule 1012 is a five-tuple filter having several wildcards. In fact, it only specifies only that the protocol used is TCP,and that the source transport port be 23399 (i.e., VoIP). In otherwords, the LRO aggregation rule 1012 states that any packet with sourcetransport 23399 and protocol TCP will be aggregated into LRO packets.When applied to the queue 1002, the LRO aggregation rule 1012 causespackets assigned to the queue 1002 to be aggregated into LRO packet ifit has source transport 23399 and protocol TCP (i.e., VoIP).

The LRO aggregation rule 1013 is also a five tuple filter having severalwild cards. It specifies only that the protocol be TCP and that thedestination transport port be 110 (i.e., POP3). In other words, the LROaggregation rule 1013 states that any packet with destination transportport 110 and protocol TCP will be aggregated into LRO packets. Whenapplied to the queue 1003, the LRO aggregation rule 1013 causes packetsassigned to the queue 1003 to be aggregated into LRO packet if it hasdestination transport port 110 and protocol TCP (i.e., POP3).

FIG. 10 illustrates only “positive” LRO aggregation rules, specifically,the illustrated LRO aggregation rules 1011-1013 are all rules thatenables LRO aggregation if the conditions set forth in the rule (i.e.,five tuple or MAC filtering) are met. Though not illustrated, someembodiments allow “negative” LRO aggregation rules that disable LROaggregation if the conditions set forth in the rules are met. In someembodiments, there can be a mixture of types of rules, i.e., some queuescan have positive LRO aggregation rules, some queues can have negativeLRO aggregation rules, some queues can have five-tuple filter thatspecifies a transport connection, and some queues can have filters basedon destination address (MAC or IP) filtering. Some embodiments alsosupport compound LRO rules that aggregate packets for multiple flows ordestination addresses.

Different embodiments implement queues for buffering incoming receivedata packets differently, and the LRO aggregation rules are applieddifferently in those different embodiments when creating LRO aggregatedpackets. FIG. 10 illustrates the application of different LROaggregation rules to different queues in the host machine, where a LROaggregation rule applied to a particular queue creates LRO aggregatedpacket by aggregating only packets assigned to that particular queue. Asmentioned above, in some embodiments, each NIC queue is assigned to acomputing resource such as a processor core or a processing thread. AnLRO aggregation rule applied to such a queue is therefore applicable tothe VM or the network stack that is being processed by that assignedcomputing resource.

For some embodiments, FIG. 11 illustrates LRO aggregation rules appliedto queues that are assigned to computing resources in a host machine1100. As illustrated, the host machine 1100 has computing resources(labeled as CPUs) 1101-1104 that are used to operate VMs 1111-1114 andnetwork stacks 1121-1124. Specifically, the computing resource 1101 isfor operating the VM 1101 and the network stack 1121, the computingresource 1102 is for operating the VM 1102 and the network stack 1122,etc. The host machine 1100 also has NIC queues 1141-1144 in a PNIC (notillustrated) for buffering incoming data packets from a physical network1190. The host machine 1100 includes a queue assignment module 1180 forassigning each incoming data packets from the physical network 1190 intoone of the queues 1141-1144 in the PNIC. The host machine 1100 alsoincludes a RSS module 1160 (receive side scaling) for assigning networktraffic from the NIC queues to the computing resources.

As illustrated, each NIC queue receives its own LRO aggregation rule andperforms LRO aggregation on data packets assigned that queue based onthe received rule. Since the network traffic from the queues aredistributed by the RSS module 1160 to one of the computing resources1101-1104, the LRO aggregation rule applied to a particular queue isapplicable to the computing resource that is selected by the RSS 1160 toreceive data packets (LRO aggregated or not) from that particular queue.In some embodiments, the RSS 1160 selects computing resources to receivedata from the queues in a manner to balance the computational loadsbetween the different computing resources.

In some embodiments, each VM and its corresponding network stack (e.g.,the VM 1111 and the network stack 1121) are handled by a same computingresource (such as a same CPU, a same CPU core, or a same processingthread of a CPU). A LRO aggregation rule applied to a queue that isassigned to a computing resource is therefore applied to the VM that isperformed by that computing resource. For example, if the traffic fromNIC queue 1141 is assigned to the computing resource 1101, then the LROaggregation rule would be producing LRO aggregated packets for the VM1111. In some embodiments, the RSS 1160 ensures that a thread of a VMstays on a CPU for the duration of a network connection session, andthus an LRO aggregation rule that enables LRO aggregation for aparticular five-tuple would remain applicable for the VM for theduration of the network connection session according to that particularfive-tuple.

FIG. 11 illustrates LRO aggregation rules that are applicable tocomputing resources in host machines rather to VMs directly. In someembodiments, each queue is directly associated with a VM, and thereforea LRO rules applied to a particular queue is always applicable to thatparticular VM. FIG. 12 illustrates LRO aggregation rules applied to NICqueues or incoming packet buffers that are each associated with a VM.

FIG. 12 illustrates a host machine 1200 that is operating VMs 1211 and1212. The host machine also has queues 1221 and 1222 for bufferingincoming data packets from a physical network 1290. The queue 1221receives incoming data packets that pass through a filter 1231 and thequeue 1222 receives incoming data packets that pass through a filter1232. The packets in the queue 1221 are aggregated by LRO aggregationmodule 1241, which enables LRO aggregation based on LRO aggregation rule1251. The packets in the queue 1222 are aggregated by LRO aggregationmodule 1242, which enables LRO aggregation based on LRO aggregation rule1252.

As illustrated, the VM 1211 has a MAC address “MAC1” and the VM 1212 hasMAC address “MAC2”. The queue 1221 receives only packets destined for VM1211, because the filter 1231 is a MAC filter that allows only datapackets destined for address “MAC1” to enter the queue 1221. Likewise,the queue 1222 receives only packets destined for VM 1212, because thefilter 1232 is a MAC filter that allows only data packets destined foraddress “MAC2” to enter the queue 1222. Consequently, the LROaggregation rule 1251 applied to the queue 1221 is applicable only topackets destined for the VM 1211, and the aggregated packet producedunder LRO aggregation rule 1251 is always destined for VM 1211,regardless of whether the LRO aggregation rule 1251 actually specifiesthe destination address (e.g., by having wild card on the “destinationIP” part of the five-tuple.) Likewise is true for the LRO aggregationrule 1252 applied to the queue 1222 and the VM 1212.

Though the example of FIG. 12 uses L2 MAC address of a VM for filteringthe incoming packets into the VM's corresponding queue, one of ordinaryskill would understand that other address schemes that uniquely addressa VM can also be used for filtering packets into the VM's queue. Forexample, some embodiments use the L3 IP address of the VM as filter forthe VM's queue.

As mentioned, in some embodiments, NIC queues (or incoming packetbuffers) are not necessarily tied to VMs. In some of these embodiments,the LRO aggregation rule or rules applicable to a NIC queue would beapplied to all incoming packets to that queue. The host machine wouldthen forward the packets from the queue (aggregated and non-aggregated)to their destination based on the destination address (e.g., destinationIP or destination MAC address) in the packet headers.

For some embodiments, FIG. 13 illustrates an LRO aggregation ruleapplied to a queue that is not bound to any specific VMs. FIG. 13illustrates a host machine 1300 that is operating VMs 1311 and 1312. Thehost machine has a queue 1320 for buffering incoming data packets from aphysical network 1390. The host machine also has software forwardingelement 1330 for forwarding packets from the queue 1320 as well as otherincoming packet buffers (not illustrated) to the VMs 1311, 1312, andothers (not illustrated). An LRO aggregation module 1340 receives a LROaggregation rule 1350 for determining whether to aggregate packets inthe queue 1320 into LRO aggregated packets. In some embodiments, the LROaggregation rule 1350 is a five-tuple flow or microflow.

As illustrated, the queue 1320 receives incoming data packets 1360 fromthe physical network 1390, the incoming data packets including somepackets with destination address “MAC1” (the MAC address of VM 1311) andsome packets with destination address of “MAC2” (the MAC address of VM1312). These packets 1360 arrive at the queue 1320, and the LROaggregation module 1340 applies the LRO aggregation rule 1350 to createaggregated packets 1371 with destination address “MAC1” and aggregatedpackets 1372 with destination address “MAC2”. Packets that do not meetthe requirement of LRO aggregation rule 1350 remain non-aggregated(non-aggregated packets 1381 for “MAC1” and non-aggregated packets 1382for “MAC2”.) The packets, whether aggregated or non-aggregated, are thenforwarded by the software forwarding element 1330 to their respectivedestinations (the VM 1311 or the VM 1312) based on the destinationaddress in the header.

In some embodiments, the LRO aggregation rule specifies only thedestination address. In other words, the LRO aggregation rule enablesLRO aggregation only for a VM having a particular address (IP address orMAC address), while packets for any other VMs will not be aggregated.For some embodiments, FIG. 14 illustrates an LRO aggregation rule 1450that specifies only a destination address for the queue 1320 in the hostmachine 1300.

The LRO aggregation rule 1450 is a rule that specifies that LROaggregation is to take place for packets with destination MAC address“MAC1”, while all other packets (i.e., packets with other destinationMAC address) will not be aggregated. Consequently, all packets (packets1471) being forwarded to the VM 1311 by the software forwarding element1330 are LRO aggregated, and packets being forwarded to the VM 1312(packets 1472) by the software forwarding element 1330 are not LROaggregated.

In some embodiments, each particular LRO aggregation rule is applied notonly to a one particular queue or incoming packet buffer, but is insteadapplied to all incoming data packets stored in all queues. Furthermore,multiple LRO aggregation rules are actively simultaneously to aggregateLRO packets under different rules, and the LRO operations of at leastone of the queues (or some or all of the queues) are governed bymultiple LRO aggregation rules.

FIG. 15 illustrates a host machine 1500 in which multiple LROaggregation rules are being applied across different NIC queues orincoming packet buffers. As illustrated, the host machine 1500 isoperating VMs 1511-1514, each of which receives data packets from anetwork stack (not illustrated), which in turn receive data from one ofthe queues 1541-1543 for buffering incoming data packets. These queuescan be NIC queues in a PNIC (not illustrated) of the host machine 1500,or a software implemented data buffers managed by a virtualizationsoftware (not illustrated). FIG. 15 also illustrates a memory storage1530, which can be one or more memory storage devices storing thecontents of the queues. In some embodiments, the retrieval of a datapacket (whether aggregated or not) from a particular queue isaccomplished by using an address pointer maintained by the particularqueue to read a block data stored in the memory storage 1530. One ofordinary skill would realize that, for some embodiments, such a memorystorage device can be used to implement some or all of the queues orincoming packet buffers discussed in Sections I and II.

The host machine 1500 has three different LRO aggregation rules1551-1553 that are applied to the queues 1541-1543. Each of the LROaggregation rules 1551-1353 is applied to all three queues 1541-1543.Each LRO aggregation rule has a different effect on different queuesdepending on the packets being held in each queue.

The LRO aggregation rule 1551 is a 5-tuple rule that does not specify aspecific destination address (i.e., having wild cards in destinationIP). The rule 1551 therefore affects all VMs and all queues, and bothqueues 1541 and 1543 have LRO aggregated packets under rule 1551(aggregated packets 1561, 1562, and 1563). However, since the rule 1551does require that the destination transport port be “110” and theprotocol be “TCP”, any packet that does not have the matching transportport ID or protocol required by the five tuple in 1551 will not beaggregated under this rule.

The LRO aggregation rule 1542 is a MAC filter, it enable LRO aggregationonly for the VM with MAC address “MAC2”. The rule 1552 therefore affectsonly a queue that is holding data packets destined for MAC address“MAC2” (i.e., the VM 1512). In the example of FIG. 15, only the queue1542 is holding packets destined for VM 1512, and therefore only thequeue 1542 has LRO aggregated packet created under the rule 1552.Furthermore, in this example, the queue 1542 holds only packets destinedfor the VM 1512, and therefore all of the data packets in the queue 1542are aggregated under the rule 1552 (aggregated packets 1564 and 1565).

The LRO aggregation rule 1553 is a completely specified five-tuplefilter with a specified destination IP address “10.10.3.1”. The rule1553 therefore affects only a queue that is holding data packetsdestined to IP address “10.10.3.1” (i.e., the VM 1513). In the exampleof FIG. 15, only the queue 1543 is holding packets destined for the VM1513, and therefore only the queue 1543 has LRO aggregated packetcreated under the rule 1553 (aggregated packet 1566). Furthermore, sincethe rule 1553 also requires that the protocol be “TCP/UDP”, the source“192.16.10.2”, the source transport port ID be “1111”, the destinationtransport port ID be “2222”, any packets failing to meet the theserequirement will not be aggregated under the rule 1553.

For some embodiments, FIG. 16 conceptually illustrates a process 1600for applying LRO aggregation rules to packets in NIC queues or incomingpacket buffers. In some embodiments, the process 1600 is performed by aPNIC for each of its NIC queues. In some embodiments, the process 1600is performed by a virtualization software implementing LRO aggregationin its software implemented incoming packet buffers.

The process 1600 starts when it receives (at 1610) a packet from thephysical network. The process then determines (at 1620) whether thepacket is for this queue. In some embodiments, the process applies adestination address filter (e.g., a MAC filter) that allows only packetswith certain destination address or addresses into the queue. In someembodiments, the process applies other criteria, such performing hashingto determine whether the incoming data packet is to be assigned to aparticular CPU that is assigned to the queue. If the packet is not forthis queue, the process ignores (at 1625) the packet and let the packetbe assigned to one of the other queues or incoming packet buffer and theprocess 1600 ends. If the packet is for this queue, the process proceedsto 1630.

At 1630, the process determines whether LRO aggregation is enabled forthis packet. For a queue that uses five-tuple microflow as LROaggregation rule to determine whether to perform LRO aggregation, theprocess examines whether the incoming packet meets the requirement ofthe five tuple. For a queue that uses another type of LRO aggregationrule (such as MAC filtering) the process examines the packet under theother criteria to determine whether to perform LRO aggregation. If thepacket meets the requirement of the LRO aggregation rule, the processproceeds to 1640. If the packet does not meet the requirement of the LROaggregation rule, the process proceeds to 1635.

At 1635, the process passes or sends the packet onto the VM withoutaggregation. In some embodiments, the process notifies (e.g., byinterrupt) the processor core of the host machine to let it know thatthat a packet is ready to be retrieved. In some embodiments, the packetis stored in a memory area awaiting retrieval, and the process uses anAPI to notify the host machine processor core of the memory location ofthe data packet. The process 1600 then ends.

At 1640, the process aggregates or adds the received packet into acurrent LRO aggregated packet that is still being aggregated. Theprocess then determines (at 1650) if the aggregation of the current LROaggregated packet is complete. In some embodiments, the process comparesthe size of the LRO aggregated packet against a threshold size (usuallylarger than the MSS of TCP or MTU of Ethernet) to determine if the LROaggregated packet is large enough for delivery/retrieval. If the LROaggregated packet is complete and ready for retrieval, the processproceeds to 1660. If the LRO aggregated packet is incomplete and canaggregate more incoming received packet, the process proceeds to 1655 tocontinue aggregation and ends.

At 1660, the process passes or sends the aggregated packet onto the VM.In some embodiments, the process notifies (e.g., by interrupt) theprocessor core of the host machine to let it know that that a packet isready to be retrieved. In some embodiments, the packet is stored in amemory area awaiting retrieval, and the process uses an API to notifythe host machine processor core of the location in memory of the LROaggregated packet. The process 1600 then ends.

III. Snooping Maximum Segment Size

As mentioned, some embodiments turn off LRO operations on VMs that areforwarding packets, partly because packets that are to be forwarded mustcomply with a maximum size requirement. Creating LRO aggregated packetsthat exceed the maximum size limit (e.g., Ethernet MTU) would requiredownstream segmentation operations (e.g., TSO operations) orfragmentation to dissolve the aggregated packets into smaller segments.However, some embodiments do perform LRO aggregation on packets beingforwarded by a VM. Some of these embodiments then segment the LROaggregated packet according to the Maximum Segment Size (MSS) of the TCPprotocol before forwarding the segmented packets to their destination.

In some embodiments, for packets being forwarded through a VM with LROhaving being turned on for the flows being handled by the VM, the VM inits forwarding path can snoop on the TCP traffic for MSS and maintainstateful table for these flows and mark the large packet for TSOprocessing based upon the MSS that it snooped. Doing so avoids breakingthe OSI model and still takes advantage of the hardware assist/offloadthat is available in PNIC (for tasks such as packet aggregation andsegmentation), and the VM does not have to do fragmentation on thepacket in the forwarding path because of maximum transmission unit (MTU)limitation.

The MSS is the largest amount of data, specified in bytes, that TCP iswilling to receive in a single segment. For best performance, someembodiments set the MSS small enough to avoid IP fragmentation, whichcan lead to packet loss and excessive retransmissions. Some embodimentsannounce MSS when the TCP connection is established. In some of theseembodiments, MSS is derived from the MTU size of the data link layer ofthe networks to which the sender and receiver are directly attached. Insome embodiments, the MSS is set to be smaller than the MTU to ensurethat a TCP segment complying with the MSS size requirement at TCP layerwould not be further segmented or fragmented at Ethernet/data linklayer.

The MSS is a parameter found in the TCP header of only certain types ofpackets (e.g., a TCP Syn/Ack packet), not just any TCP header. The MSSparameter is typically set by the TCP layer of the protocol stack thatoriginates the connection and then used by the TCP layer of the protocolstack that terminates the connection. The MSS parameter is typically notavailable to the forwarding VM, since the network stack of theforwarding VM does not process forwarded traffic at TCP layer or above.

Some embodiments therefore snoop the packets being forwarded for its MSSparameter before using the snooped MSS parameter to perform TransmitSegmentation Offload (TSO) operation. FIGS. 17a-b conceptuallyillustrates the snooping of MSS parameter and the use of the snooped MSSparameter for packet segmentation on packets being forwarded by a VM. Inthe example, the packets are being forwarded by a VM 1710 of a hostmachine 1700 from a source network node 1792 to a destination networknode 1794 over a physical network 1790. The host machine 1700 is alsooperating a virtualization software 1720 for hosting the VM 1710 andother VMs (not illustrated).

FIG. 17a illustrates the snooping of the MSS parameter in fouroperations labeled ‘1’, ‘2’, ‘3’, and ‘4’. In operation ‘1’, the sourcenode 1792 sends a packet 1751 to the host machine 1700. The packet 1751contains a TCP syn/ack packet of a particular TCP connection, whoseheader includes a MSS parameter. In operation ‘2’, the packet 1751reaches the host machine 1700, and the host machine extracts MSSparameter from the header of the TCP syn/ack packet and stores theextracted MSS. In some embodiments, the extraction of MSS isaccomplished by the network stack of the VM 1710, which in someembodiments is considered to be part of the virtualization software ofthe host, while in some other embodiments the network stack of the VM1710 is considered to be part of the VM itself. In operation ‘3’, the VM1710 forwards the packet 1751 (including the TCP syn/ack packet) throughthe virtualization software 1720. In operation ‘4’, the packet 1751reaches the destination node 1794. Though not illustrated, in someembodiments, the packet 1751 can be an LRO aggregated packet.

FIG. 17b illustrates LRO aggregation and the use of the snooped MSS toperform segmentation (TSO) operations on the LRO aggregated packets bythe host machine 1700 in five operations labeled ‘5’, ‘6’, ‘7’, ‘8’, and‘9’. In operation ‘5’, the source node 1792 sends packets 1752 to thehost machine 1700 destined for the forwarding VM 1710. In operation ‘6’,the host machine 1700 receives the packets 1752 at an incoming packetbuffer or NIC queue 1730 and perform LRO aggregation based on anapplicable LRO aggregation rule. The LRO aggregation produced anaggregated packet 1753, which is larger in size than allowed by MSS. Inoperation ‘7’, LRO aggregated packet 1753 reaches the VM 1710 and isforwarded by the VM. In operation ‘8’, the host machine performssegmentation at a segmentation module 1740, which uses the earlierextracted MSS parameter to segment the LRO aggregated packet 1753 intosegmented packets 1754 that are each smaller or equal in size than theextracted MSS. In some embodiments, the syn/ack packet from which theMSS parameter is snooped is for establishing a particular TCPconnection, and the MSS parameter is therefore only used for segmentingpackets of that particular TCP connection.

In some embodiments, packets being forwarded by a VM do not traverse allthe way up in the network stack of the VM, but are rather handled atlower layers of the network stack only. Specifically, for packets beingforwarded by a VM, some embodiments only handle the network protocol ofthe link layer (Ethernet layer) and the Internet layer (IP layer), butnot for the TCP transport layer and/or above. Since MSS parameter is inTCP layer header but the forwarded packet is never processed by theforwarding VM's TCP layer, some embodiments snoop and extract the MSSparameter from the TCP header when the lower layers of the network stackforwards the syn/ack packet.

FIG. 18 illustrates the snooping and extraction of the MSS parameter atlayers of network stack lower than TCP layer. Specifically, FIG. 18illustrates the extraction of the MSS parameter from packet 1751 by anetwork stack 1720 of the VM 1710 in the host machine 1700. The hostmachine 1700 also includes a NIC 1760 for interfacing with the physicalnetwork. The network stack 1720 includes a link layer (Ethernet) 1721,an Internet layer (IP) 1722, a transport layer (TCP) 1723, and anapplication layer 1724.

As illustrated, the packet 1751 is a packet with several layers ofencapsulation, in which successive higher layer header and payload areencapsulated as a lower layer payload along with a lower layer header.The packet 1751 encapsulates a TCP layer packet under IP layer andEthernet layer, and consequently includes an Ethernet layer header, anIP layer header, and a TCP layer header and TCP payload. An MSSparameter is included in the TCP header since the encapsulated TCP layerpacket is a TCP syn/ack packet. In some embodiments, the packet 1751 canbe an LRO aggregated packet.

FIG. 18 illustrates the snooping and extraction of MSS parameter in fouroperations labeled ‘1’, ‘2’, ‘3’, and ‘4’. In some embodiments, theoperations ‘1’ through ‘4’ of FIG. 18 correspond to the operations ‘1’through ‘4’ of FIG. 17a . In operation ‘1’, the packet 1751 arrives atthe host machine 1700, queued and delivered by the NIC 1760 to thenetwork stack 1720. The network stack 1720 processed the protocols atthe link layer 1721 and the internet layer 1722 but not at any higherlayer (e.g., processed parameters in the link layer header and the IPlayer header but not headers of other layers). In operation ‘2’, thenetwork stack 1720 performs snooping and extracts the MSS parameter fromthe TCP layer header without actually handling the TCP layer protocols(e.g., without processing the TCP layer header). In operation ‘3’, thenetwork stack 1720 forwards the packet 1751 through the NIC 1760. Inoperation ‘4’, the packet 1751 is forwarded out of the host machine 1700to its destination.

In some embodiments, the segmentation operation that uses the extractedMSS parameter is performed by the NIC 1760 of the host machine 1700. Insome of these embodiments, the NIC 1760 is a PNIC that is performs boththe aggregation operation (LRO) and the segmentation (TSO) within itsown hardware without consuming CPU cycles at the host machine 1700. Insome embodiments, the PNIC receives the MSS parameter from the networkstack 1720 as a metadata that accompanies a LRO aggregated packet.

For some embodiments, FIG. 19 illustrates the segmentation (TSO)operation by the PNIC based on a MSS parameter extracted by the networkstack 1720 in the host machine 1700. As illustrated, the NIC 1760 in thehost machine 1700 is a PNIC. It performs LRO aggregation on the incomingdata packets 1752 and TSO segmentation the forwarded packet into thesegmented packets 1754. In some embodiments, the PNIC 1760 performs LROaggregation on the incoming packets 1754 at a NIC queue 1730 based on aLRO aggregation rule as described above in Sections I and II.

FIG. 19 illustrates the LRO and TSO operations in five operationslabeled ‘5’, ‘6’, ‘7’, ‘8’, and ‘9’. In some embodiments, the operations‘5’ through ‘9’ of FIG. 19 correspond to the operations ‘5’ through ‘9’of FIG. 17b . In operation ‘5’, the incoming packets 1752 arrive at thehost machine 1700 from the source node 1792 and reach the PNIC 1760,where the packets are buffered at the queue 1730. In operation ‘6’, thePNIC 1760 performs LRO aggregation according to a LRO aggregation ruleapplied to the packets at the queue 1730, which in turn produces an LROaggregated packet 1753. In operation ‘7’, the network stack 1720 of theVM 1710 handles the network protocols for the packet 1753 at the linklayer 1721 and the internet layer 1720, and then forwards the LROaggregated packet to the PNIC 1760 along with the MSS parameter that waspreviously extracted and stored. In some embodiments, the MSS parameteris passed to the PNIC 1760 as metadata 1757 accompanying the aggregatedpacket 1753. In operation ‘8’, the PNIC 1760 performs segmentation onthe LRO aggregated packet 1753 according to the MSS parameter receivedfrom the network stack 1720. In operation ‘9’, the PNIC sends outsegmented packets 1754 to the physical network, each segmented packet issmaller in size than the MSS.

As mentioned, in some embodiments, a VM in its forwarding path can snoopon the TCP traffic for MSS and maintains stateful tables for these flowsand mark the large packet for TSO processing based upon the MSS that itsnooped. A stateful VM keeps track of the state of network connections(such as TCP streams or UDP communication) and is able to holdsignificant attributes of each connection in memory. These attributesare collectively known as the state of the connection, and may includesuch details as the IP addresses and ports involved in the connectionand the sequence numbers of the packets traversing the connection.Stateful inspection monitors incoming and outgoing packets over time, aswell as the state of the connection, and stores the data in dynamicstate tables. This cumulative data is evaluated, so that context thathas been built by previous connections and/or previous packets belongingto the same connection would be available.

FIG. 20 conceptually illustrates the maintenance of stateful tables by aforwarding VM while snooping for MSS. Specifically, FIG. 20 illustratesthe VM 1710 in the host machine 1700 snooping for MSS and maintainingstateful tables for three different TCP connections A, B, and C. Eachstateful table of a TCP connection is in turn used to perform TSOsegmentation operations on the traffic of that TCP connection.

As illustrated, the host machine 1700 is receiving packets 2010 fromvarious source nodes. These received packets 2010 includes packets indifferent TCP connections A, B, C (rectangles labeled A, B, and C in2010). In some embodiments, each TCP connection includes traffic in bothdirections (such as SYN and ACK packets) such that packets of aparticular connection can come from either end of the TCP connection.

The LRO aggregation module 1730 applies LRO aggregation rules on thereceived packets 2010 and produces packets 2020. Since all of thereceived packets 2010 are to be processed and forwarded by the VM 1710,some embodiments assign all of the received packets 2010 to a same NICqueue in the PNIC, and the LRO module 1730 represents the LROaggregation operation that takes place in one NIC queue. In someembodiments, packets destined for a same VM can be assigned to differentqueues, and the LRO aggregation module 1730 represents LRO aggregationoperations performed at different queues of the NIC. In someembodiments, the LRO aggregation rules are flow based rules that targetone or more TCP connections, i.e., only packets belonging to certain TCPconnections will be aggregated. Though not indicated in the figure, someof the packets 2020 outputted by the LRO aggregation module 1730 areaggregated as they satisfy the criteria specified by the LRO rule, whilesome of the packets are not aggregated as they do not meet the LRO rule.

The packets 2020 are delivered to the VM 1710 and its correspondingnetwork stack for processing and forwarding. The VM 1710 operates asnooper module 2050 that snoops the content of the packets 2020 beingforwarded. The snooper 2050 keeps track of the states of each TCPconnection in stateful tables 2041, 2042, and 2043. Among the stateinformation being maintained is the MSS parameter of each TCPconnection. Specifically, “MSS A” is the MSS of TCP connection A, “MSSB” is the MSS of TCP connection B. “MSS C” is the MSS of TCP connectionC. The VM 1710 marks some of the packets 2020 as requiring TSOsegmentation operation, and the MSS parameters of the corresponding TCPconnections are passed to the TSO module 1740 along with the markedpackets as metadata. The marked packets are then sent to the TSO module1740 as packets 2030.

The marked packets 2030 forwarded by the VM 1710 reaches the TSO module1740, which segments some or all of the packets according to the MSSparameter of the TCP connection that the packet belongs to. For example,the packet 2021 belongs to TCP connection A, and the TSO module 1740would segment it according to MSS A (i.e., each segmented packet is lessthan or equal to MSS A). Likewise, the packet 2022 would be segmentedaccording to the MSS of connection B, and the packet 2023 would besegmented according to the MSS of connection C. In some embodiments,each packet requiring TSO segmentation would arrive at the TSO module1740 with a metadata that indicates its MSS (i.e., the MSS of its TCPconnection), and the TSO module 1740 then segments the packet based onthe MSS embedded in the metadata of the packet. The segmented packetsare then delivered to their destinations as packets 2040.

In some embodiments, FIG. 21 conceptually illustrates a process 2101 forsnooping MSS parameter and a process 2102 for performing segmentation onLRO aggregated packets based on the snooped MSS parameter. The process2102 uses the MSS parameter snooped by the process 2101 for performingsegmentation. In some embodiments, the processes 2101 and 2102 areperformed by a host machine having a PNIC. In some embodiments, the twoprocesses are performed in the same host machine in parallel.

The process 2101 starts when it receives (at 2110) an incoming packetfrom a network. The process then determines (at 2120) if MSS informationis available in the packet. In some embodiments, not all packets haveTCP headers, and not all TCP headers specify MSS. Some embodimentsexamine the packet for a particular type of TCP header (e.g., syn/ack)to determine if MSS parameter is available. If MSS parameter isavailable, the process 2101 proceeds to 2125 to extract and store theMSS and ends. If MSS parameter is not available, the process 2101 ends.The MSS parameter, if extracted, is made available to process 2102.

The process 2102 starts whenever there are incoming packets from thenetwork. The process 2102 performs (at 2160) LRO aggregation on theincoming packets. In some embodiments, this aggregation is enabledaccording to LRO aggregation rules such as five-tuple microflows asdiscussed above in Section II. The process then performs (at 2170) TSOsegmentation on outgoing packets based on the extracted MSS. In someembodiments, an extracted MSS is specific to a TCP connection, and theprocess 2102 performs segmentation on a packet based on the MSS of theTCP connection that the packet belongs to. After performing segmentationon the aggregated packet, the process 2102 ends.

IV. Electronic System

Many of the above-described features and applications are implemented assoftware processes that are specified as a set of instructions recordedon a computer readable storage medium (also referred to as computerreadable medium). When these instructions are executed by one or moreprocessing unit(s) (e.g., one or more processors, cores of processors,or other processing units), they cause the processing unit(s) to performthe actions indicated in the instructions. Examples of computer readablemedia include, but are not limited to, CD-ROMs, flash drives, RAM chips,hard drives, EPROMs, etc. The computer readable media does not includecarrier waves and electronic signals passing wirelessly or over wiredconnections.

In this specification, the term “software” is meant to include firmwareresiding in read-only memory or applications stored in magnetic storage,which can be read into memory for processing by a processor. Also, insome embodiments, multiple software inventions can be implemented assub-parts of a larger program while remaining distinct softwareinventions. In some embodiments, multiple software inventions can alsobe implemented as separate programs. Finally, any combination ofseparate programs that together implement a software invention describedhere is within the scope of the invention. In some embodiments, thesoftware programs, when installed to operate on one or more electronicsystems, define one or more specific machine implementations thatexecute and perform the operations of the software programs.

FIG. 22 conceptually illustrates an electronic system 2200 with whichsome embodiments of the invention are implemented. The electronic system2200 can be used to execute any of the control, virtualization, oroperating system applications described above. The electronic system2200 may be a computer (e.g., a desktop computer, personal computer,tablet computer, server computer, mainframe, a blade computer etc.),phone, PDA, or any other sort of electronic device. Such an electronicsystem includes various types of computer readable media and interfacesfor various other types of computer readable media. Electronic system2200 includes a bus 2205, processing unit(s) 2210, a system memory 2225,a read-only memory 2230, a permanent storage device 2235, input devices2240, and output devices 2245.

The bus 2205 collectively represents all system, peripheral, and chipsetbuses that communicatively connect the numerous internal devices of theelectronic system 2200. For instance, the bus 2205 communicativelyconnects the processing unit(s) 2210 with the read-only memory 2230, thesystem memory 2225, and the permanent storage device 2235.

From these various memory units, the processing unit(s) 2210 retrievesinstructions to execute and data to process in order to execute theprocesses of the invention. The processing unit(s) may be a singleprocessor or a multi-core processor in different embodiments.

The read-only-memory (ROM) 2230 stores static data and instructions thatare needed by the processing unit(s) 2210 and other modules of theelectronic system. The permanent storage device 2235, on the other hand,is a read-and-write memory device. This device is a non-volatile memoryunit that stores instructions and data even when the electronic system2200 is off. Some embodiments of the invention use a mass-storage device(such as a magnetic or optical disk and its corresponding disk drive) asthe permanent storage device 2235.

Other embodiments use a removable storage device (such as a floppy disk,flash drive, etc.) as the permanent storage device. Like the permanentstorage device 2235, the system memory 2225 is a read-and-write memorydevice. However, unlike storage device 2235, the system memory is avolatile read-and-write memory, such a random access memory. The systemmemory stores some of the instructions and data that the processor needsat runtime. In some embodiments, the invention's processes are stored inthe system memory 2225, the permanent storage device 2235, and/or theread-only memory 2230. From these various memory units, the processingunit(s) 2210 retrieves instructions to execute and data to process inorder to execute the processes of some embodiments.

The bus 2205 also connects to the input and output devices 2240 and2245. The input devices enable the user to communicate information andselect commands to the electronic system. The input devices 2240 includealphanumeric keyboards and pointing devices (also called “cursor controldevices”). The output devices 2245 display images generated by theelectronic system. The output devices include printers and displaydevices, such as cathode ray tubes (CRT) or liquid crystal displays(LCD). Some embodiments include devices such as a touchscreen thatfunction as both input and output devices.

Finally, as shown in FIG. 22, bus 2205 also couples electronic system2200 to a network 2265 through a network adapter (not shown). In thismanner, the computer can be a part of a network of computers (such as alocal area network (“LAN”), a wide area network (“WAN”), or an Intranet,or a network of networks, such as the Internet. Any or all components ofelectronic system 2200 may be used in conjunction with the invention.

Some embodiments include electronic components, such as microprocessors,storage and memory that store computer program instructions in amachine-readable or computer-readable medium (alternatively referred toas computer-readable storage media, machine-readable media, ormachine-readable storage media). Some examples of such computer-readablemedia include RAM, ROM, read-only compact discs (CD-ROM), recordablecompact discs (CD-R), rewritable compact discs (CD-RW), read-onlydigital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a varietyof recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.),flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.),magnetic and/or solid state hard drives, read-only and recordableBlu-Ray® discs, ultra density optical discs, any other optical ormagnetic media, and floppy disks. The computer-readable media may storea computer program that is executable by at least one processing unitand includes sets of instructions for performing various operations.Examples of computer programs or computer code include machine code,such as is produced by a compiler, and files including higher-level codethat are executed by a computer, an electronic component, or amicroprocessor using an interpreter.

While the above discussion primarily refers to microprocessor ormulti-core processors that execute software, some embodiments areperformed by one or more integrated circuits, such as applicationspecific integrated circuits (ASICs) or field programmable gate arrays(FPGAs). In some embodiments, such integrated circuits executeinstructions that are stored on the circuit itself.

As used in this specification, the terms “computer”, “server”,“processor”, and “memory” all refer to electronic or other technologicaldevices. These terms exclude people or groups of people. For thepurposes of the specification, the terms display or displaying meansdisplaying on an electronic device. As used in this specification, theterms “computer readable medium,” “computer readable media,” and“machine readable medium” are entirely restricted to tangible, physicalobjects that store information in a form that is readable by a computer.These terms exclude any wireless signals, wired download signals, andany other ephemeral signals.

While the invention has been described with reference to numerousspecific details, one of ordinary skill in the art will recognize thatthe invention can be embodied in other specific forms without departingfrom the spirit of the invention. In addition, a number of the figures(including FIGS. 2, 16, and 21) conceptually illustrate processes. Thespecific operations of these processes may not be performed in the exactorder shown and described. The specific operations may not be performedin one continuous series of operations, and different specificoperations may be performed in different embodiments. Furthermore, theprocess could be implemented using several sub-processes, or as part ofa larger macro process. Thus, one of ordinary skill in the art wouldunderstand that the invention is not to be limited by the foregoingillustrative details, but rather is to be defined by the appendedclaims.

What is claimed is:
 1. A method for operating a host machine in avirtualized network environment, the host machine hosting a plurality ofvirtual machines, the method comprising: specifying a set of rules fordetermining whether or not to aggregate packets at a plurality ofincoming packet buffers, each rule in the set of rules controlling apacket aggregation operation in a corresponding incoming packet bufferof the plurality of incoming packet buffers, wherein a first rulespecifies not aggregating packets for a first set of packets at a firstincoming packet buffer and a second rule specifies aggregating packetsfor a second set of packets at a second incoming packet buffer, whereinpacket header contents distinguish the first set of packets from thesecond set of packets; forwarding a non-aggregated packet from the firstincoming packet buffer to a first virtual machine of the plurality ofvirtual machines, wherein the first virtual machine forwards thenon-aggregated packet; and forwarding an aggregated packet from thesecond incoming packet buffer to a second virtual machine of theplurality of virtual machines, wherein the second virtual machineterminates the aggregated packet.
 2. The method of claim 1 furthercomprising specifying a five-tuple identification for the particulartransport connection.
 3. The method of claim 1 further comprisingspecifying a protocol identifier, a source IP address, a destination IPaddress, a source transport identifier, and a destination transportidentifier for the particular transport connection.
 4. The method ofclaim 1 wherein the first rule specifies not aggregating packetsassociated with a particular transport connection at the first incomingbuffer.
 5. The method of claim 4, wherein the first rule furtherspecifies packets that belong to a different transport connection willbe aggregated.
 6. The method of claim 4, wherein the first rulespecifies a five-tuple identification with at least one wild card, theparticular transport connection being associated with the five-tupleidentification.
 7. The method of claim 1, wherein the plurality ofincoming packet buffers are queues in a network interface controller(NIC).
 8. The method of claim 1, wherein the second rule specifiesaggregating packets having a particular destination MAC address, thedestination MAC address being an address of the second virtual machine.9. A computing device serving as a host machine in a virtualized networkenvironment, the computing device comprising a set of processing unitsexecuting a computer program comprising sets of instructions for:executing a plurality of virtual machines; specifying a set of rules fordetermining whether or not to aggregate packets at a plurality ofqueues, each rule in the set of rules controlling a packet aggregationoperation in a corresponding queue of the plurality of queues, wherein afirst rule specifies not aggregating packets for a first set of packetsat a first queue and a second rule specifies aggregating packets for asecond set of packets at a second queue, wherein packet header contentsdistinguish the first set of packets from the second set of packets;forwarding a non-aggregated packet from the first queue to a firstvirtual machine of the plurality of virtual machines, wherein the firstvirtual machine forwards the non-aggregated packet; and forwarding anaggregated packet from the second queue to a second virtual machine ofthe plurality of virtual machines, wherein the second virtual machineterminates the aggregated packet.
 10. The computing device of claim 9,wherein the first rule specifies that packets belonging to a particulartransport connection are not aggregated at the first queue.
 11. Thecomputing device of claim 10, wherein the particular transportconnection is a first transport connection, the computer program furthercomprising sets of instructions for aggregating packets belonging to asecond transport connection into an aggregated packet at a third queue.12. The computing device of claim 9, wherein determining whether toaggregate packets for the particular transport connection furthercomprises analyzing a five-tuple identification for the particulartransport connection.
 13. The computing device of claim 12, wherein thefive-tuple identification comprises at least one wild card.
 14. Thecomputing device of claim 12, wherein the five tuple identificationcomprises a protocol identifier, a source IP address, a destination IPaddress, a source transport identifier, and a destination transportidentifier for the particular transport connection.
 15. The computingdevice of claim 9 further comprising sets of program instructions fordirecting an assignment of packets received from a physical network toone of the first and second queues, wherein a packet assigned to thefirst queue is for one or more first virtual machines and a packetassigned to the second queue is for one or more second virtual machines.16. A method for controlling large receive offload (LRO) operations at anetwork interface controller (NIC), the method comprising: receiving, ata plurality of NIC queues, a plurality of incoming packets from anetwork, wherein each of the incoming packets contain transportconnection information; specifying a set of filters, wherein each of thefilters comprises a five-tuple specifying particular transportconnection information and a rule determining whether or not toaggregate an incoming packet containing the particular transportconnection information at a NIC queue; for each of the received incomingpackets: matching the transport connection information of the receivedincoming packet with the corresponding transport connection informationspecified by the five-tuple in a filter of the plurality of filters;aggregating, at the NIC queue, the incoming packet according to the ruleof the filter; and retrieving the packets from the NIC queues.
 17. Themethod of claim 16, wherein the five-tuple comprises a protocolidentifier, a source IP address, a destination IP address, a sourcetransport identifier, and a destination transport identifier.
 18. Themethod of claim 16 further comprising operating a plurality of virtualmachines and forwarding the retrieved packets to the virtual machines.19. The method of claim 16, wherein a five-tuple includes at least onewild card.